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i. Foreword 


The report presented herein comprises a summary of the development, 
implementation, and measured outcomes over the first year (October, 1992 - October, 
1993) of the Evaluation Consultation Program, an initiative funded by the Trillium 
Foundation and sponsored by the Social Planning and Research Council of Hamilton- 
Wentworth. 


This comprehensive Final Evaluation Report will review, indepth, program 
outcomes relative to measured levels of achieving program objectives and participant 
satisfaction from participating community organizations. More specific conclusions and 
recommendations will also be examined, with implications for future program 
development. 
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1.0 Background 


The inability to develop and implement systematic and self-defined evaluation 
processes is a long-standing problem within human service organizations. Being able, 
as an administrator, to draw from an information system that describes "What works" and 
“what doesn't" is a skill which has historically been an oversight on the part of many 
organizations in this service sector. 


The ability of these agencies to effectively plan services and make informed 
decisions has often been hampered by a limited knowledge base of "outcomes" 
information. 


A January, 1990 survey by the Social Planning and Research Council (SPRC) 
concluded that the major obstacles to effective planning were related to a general lack of 
skills and knowledge about client-based information systems (databases), the micro- 
computer software programs most appropriate for managing these systems and valid (and 
practical) methods for evaluating outcomes in the context of program goals and 
objectives. The survey also identified at least 35 local agencies interested in utilizing the 
services of a specialized program of evaluation consultation, if it was available locally. 


An effective evaluation consultation program, according to survey respondents, 
would include a "one-on-one" consultation component to provide direct assistance to 
agency personnel on an agency-requested basis. Additionally, in the longer term, the 
program could also serve in a resource development and networking role across the local 
community of human services in Hamilton-Wentworth. 


Based on survey results and, in keeping with the long history the SPRC has had 
in program evaluation in this community, a proposal for a time-limited consultation service 
was envisioned. Its principle goal would be the provision of consultation services to local 
agencies requiring technical assistance in the self-evaluation of existing programs, 
services, and community development activities. 


2.0 Basis for Evaluation Consultation Program Development 


With nearly thirty years of experience in assisting the local community with 
evaluation of programs and services, the SPRC has developed and promoted a unique 
approach to evaluation. This model functions based upon a solid set of principles rooted 


in both research and community development fields: 


iF 


It is important to recognize that effective planning in any organization, by 
necessity, depends upon effective evaluation of goals and objectives. Thus, 
the purpose of evaluation is to gather information about the organization 
which will be useful in determining the extent to which these goals and 
objectives have been met; 


Methods for conducting evaluations should be practical and clearly 
understood by members of the organization. Thus, it is useful to consult 
organization members in developing any evaluation design, taking into 
consideration their interests and knowledge of the organization; 


Evaluation provides information which inform choices about the future of the 
organization and should be seen as contributing to opportunities for change. 


The evaluation process should be: 


i) constructive in nature, and useful to organization members in re- 
directing the efforts of the organization or revising its vision. 


self-evaluative; that is, members of the organization own the process 
and its results. The evaluation must first serve their need for 
measures of effectiveness and success with the goals for the 
organization established by members. 


participatory by design and include in the process useful opportun- 
ities for input from all elements of the organization and those affected 
by its work. 


iv) routine and regular; evaluation must be conducted on a regularly- 


scheduled basis rather than when convenient or in response to crisis. 


3.0 Program Objectives 


The Evaluation Consultation Program was created in October, 1992 to achieve the 
following objectives in its first year: 


ie To provide consultation services to local social service agencies on matters 
pertaining to: 


a) the design of evaluation-based information systems in individual 
agencies; 


b) the use of existing database programs for client records and 
evaluation-based information systems. 
Consultation services were to be provided in the following areas: 
i) the training of agency staff in evaluation techniques; 


il) the development of client-based computerized information systems 
within requesting participating agencies; 


ili) the implementation of appropriate measuring instruments for 
monitoring client progress within participating agencies; 


2) To facilitate the development of a network of social service agency staff who 
are trained in the principles of program evaluation and database manage- 
ment. 

os To produce resources (written and otherwise) for use in social service 


agencies, including an evaluation "manual" for community service adminis- 
trators. 


4.0 Tracking Program Development and Implementation 


1. Structural Overview of ECP Program Development 


The Evaluation Consultation Program was structured to meet three functional 
priorities, relating to program objectives. Figure 1 illustrates program component areas. 


Consultation Service 


Based on an as-requested, first-come-first-served format, this program component 
offered individualized agency/organization consultation regarding a myriad of evaluation 
and program measurement issues. First priority for consultation time was given to non- 
profit, social services agencies and organizations. However, other groups and services 
were also considered when time and resources permitted. 

Staffing for the Consultation Service component comprised a full-time Evaluation 
Consultant and a part-time Database Consultant. 

In terms of activities carried out as part of this service, staff were involved in 
facilitating, training, conceptual development/planning, and technical support roles, 
depending on the nature of the request. While many organizations requested only “one 
time" consultations with staff, a greater number (refer to "Section 5.0: Program Outcome 
Measures" for more details) requested more than one type of advice or support and more 
than one consultation session. Referring again to Figure 1, these staff involvements were 
principally (but not exclusively) focused on these functional areas of evaluation research: 
One-on-one staff technical development, Information Systems Development, Evaluation 
Planning, Analytical Training and Support, and Micro-Computer Applications/Support. 


a) One-on-One Staff Development (Evaluation and Measurement Techniques) 


Working with individuals, managers, researchers, planning staff, or direct service 
workers in "hands-on" evaluation activities. A majority of this assistance centred 
on the development of self-defined and self-implemented research designs and 
evaluation measurement tools for use in client outcome and satisfaction evaluation. 
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d) 


Information Systems Development 


Any comprehensive planning and evaluation system will have a good balance of 
program and client information, collected from as many valid sources as possible 
(or feasible). Given this context, information systems development is a critical tool 
for decision-making in organizations. As such, the skills and training required to 
create, manage and utilize any "database system" required this key area of 
program service. Much of this consultation work was initially discussed or 
developed (at least conceptually) with agency directors and program managers. 


Evaluation Planning 


Typically, this service component involved working with directors or managers and, 
to some extent, with agency volunteers. The main focus of discussion was the 
identification (or reiteration) of program goals and objectives which were considered 
constructive and measurable for the purpose of evaluation. Often, creating a series 
of "measurable criteria" for determining program outcomes was the first activity 
undertaken in this regard. If an information system, or client database already 
existed within the organization, the consultant would, in collaboration with agency 
Staff, review the database elements to determine effective matching between this 
information and the measurement objectives which had been determined as part 
of the evaluation planning exercise. 


Analytical Training and Support 


The consultants established training sessions to fit the measurement needs of the 
organization. Levels of training were determined based on the existing level of 
expertise (or lack thereof) in the organization. This also determined the extent to 
which ECP staff became directly involved in computer analysis and/or output 
analysis of survey results. Methodological support in the interpretation of data 
results and survey measures was also provided where deemed necessary by 
participants. 


e) Micro-Computer Applications and Support 


In addition to the Evaluation and Database Consultants, two other volunteer 
computer programmers from the Hamilton-Wentworth community served 
astechnical advisors, or "troubleshooters" on behalf of the program. For the 
most part, agency needs focused on evaluating appropriate database or 
spreadsheet software useful in the management and analysis of client- 
based data. Some agencies had requirements which were more technical 
in nature, requiring expertise in Disk Operating Systems (DOS) and other 
operational environments This is where the volunteer computer program- 
mers were very effective in assisting managers, as well as ECP consultants. 


Evaluation Network 


The primary focus of the Evaluation Network was to provide evaluation-based 
resources and learning opportunities to staff in participating agencies. Specifically, it 
would create a unique environment in which social and other human service providers 
could meet and share ideas or information about the complexities of evaluation research 
and the planning required in creating innovative, self-evaluative tools for measuring 
program effectiveness and client impacts. Participation in the Network was generally the 
result of some initial contact with an ECP consultant, either through a direct consultation 
activity or "through the agency grapevine". However, a number of other participants 
became aware of the Network either through SPRC mail-outs of promotional information 
to agencies, or through informal information dissemination across the community of 
human services in Hamilton-Wentworth. 


The year-long development of the Evaluation Network was originally conceived, 
based on a belief that participants should be directly involved in participating and building 
its activities in an "emerging community resource network". Based upon three initial 
planning workshops with interested Network participants (n=30 participants), five (5) areas 
of Network development were identified: 


a) Information and resource-sharing (amongst groups and individual partici- 
pants); 
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b) Professional development opportunities for individuals from agencies (both 
as participants and as evolving facilitators of evaluation planning activities); 

C) Development of a Community Resource Database to be accessible by any 
user in the community; 

d) Promotion and support for inter-agency collaboration (e.g., sharing ideas, 
methods, tools, instruments, and technologies); 

e) Serving as a forum for community consultation regarding agency evaluation 
and planning issues. 


These goals were to be achieved through an extensive and planned series of 
Network activities, including workshops, instructional seminars, written resource materials, 
and a Network newsletter (refer to "Section 5.0: Program Outcome Measures" for more 
information). Many of these initiatives were identified by participants at the planning 
sessions early in the program's development. 


Resource Development 


A series of written publications regarding evaluation and measurement techniques 
was planned. By the end of the first year of program operation, three separate resource 
documents had been produced, including: a manual for developing and managing 
information systems; a "how to" manual on evaluating database management systems 
and related software needs; and an conceptual discussion paper on "Evaluation 
Principles" for non-profit organizations. 


Other resources to be developed included an SPRC in-house Evaluation Resource 
Library (part of its existing computerized library database), a Network newsletter to 
highlight activities relevant to participants in the Evaluation Network, and various 
workshop materials, such as overheads and typewritten hand-outs. 

Another area of resource development involved professional development of 
volunteers from within the Evaluation Network to facilitate activities or present information 
pertaining to agency evaluation from time to time. A total of 10 Evaluation Network 
participants also volunteered their time and resources to these ECP initiatives. 


2. Implementation Analysis 


An important consideration in evaluating program development, especially in the 
first year of a new innovation (such as the Evaluation Consultation Program) is whether 
or not components of the program have actually been implemented as planned. Before 
outcome or satisfaction measures can be undertaken, the context for measuring these 
variables must be found in an examination of implementation activities. 


As part of an ongoing analysis of program implementation, the Evaluation 
Consultant monitored the growth and development of the ECP in its first year, using a 
chronological and narrative/descriptive methodology. Appendix A provides a detailed 


monthly Chronology of Key Program Activities, beginning with the program's inception in 
October, 1992 and continuing throughout the next 12-month period, to October, 1993. 


This analysis can be instructive in terms of providing insight to evaluators on the 
implementation of program objectives and, in this case, the three specific program 
components: a Consultation Service, an Evaluation Network, and Resource Development. 
The Final Evaluation Report will incorporate a more in-depth interpretation of the 
implementation analysis and the issues identified for future program development. 


10. 
5.0 Program Outcome Measures 


This section provides an analysis of measured outcomes relating to each of the 
three program objectives originally identified in the development of the proposal for the 
Evaluation Consultation Program (refer back to "Section 3: Program Objectives"). 


The degree to which program activities and outcomes were effective in achieving 
these objectives requires consideration of a number of program and client-based 
variables. Within the context of this Program Summary Report, program and client 


outcome measures have been limited to data (anecdotal, attendance records, participation 
lists) maintained by the program consultants over the course of the 12-month program. 


Client self-reporting on outcomes and levels of satisfaction with the program are 
considered critical to any meaningful evaluation of a program. Thus, these measures are 
included in the companion Final Evaluation Report. By combining and matching program- 


generated data with client-based data, the most valid (and meaningful) interpretation of 
program effectiveness can be realized. 


1. Consultation Service Outcomes 


A total of 34 community organizations from the Hamilton-Wentworth region entered 
into direct, one-on-one consultation with staff from the Evaluation Consultation Program 
(ECP) during the 12-month period of program implementation (refer to Appendix B). This 
number reflected, almost exactly, the same number of requesting organizations or 
agencies self-identifying with this need in responding to the Evaluation Needs Survey 
(SPRC, 1990). In the study, n=35 local agencies identified an interest in "...utilizing the 
services of a specialized program of evaluation consultation, if it was available locally".(1) 
The degree to which the actual ECP client caseload of 34 agencies resembled the (n=35) 
sub-sample from the needs survey could not be determined with accuracy. 


In terms of the total number of agency personnel to actually consult with either the 
Evaluation or Database Consultant, 72 persons utilized some portion of ECP staff time 
in direct, one-on-one consultation activities. An average of 2 persons per agency met with 


(He 
a consultant at least once over the 12-months of the program (refer to Table 4.0). 


Consultation Service Utilization Profile 


The following is a descriptive analysis of key variables used to measure the 
utilization of resources by participating organizations within the Consultation Service 
component of the ECP. Variables in the analysis included: (1) organizational size; (2) 
type of organization; (3) services delivered by organization; (4) total number of personnel 
per organization using the ECP; (5) initial organizational contact; (6) total number of 
consultation activities undertaken with each organization; and, (7) total number of 
organizations to participate in specific consultation activities. 


a) Organizational Size 


Table 1.0 illustrates a breakdown of the total number of organizations involved with 
the Consultation Service by the approximate size of the organization. 


Table 1.0: Organizational Size 
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11 to 15 Staff Persons 
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21 to 25 Staff Persons 
More than 25 Staff Persons 


TOTAL 


16 to 20 Staff Persons eee 


te. 
The average size of consulting organizations was between 6-10 staff persons, 
however, the range of organizational size was between 1 and 25-plus personnel. 
Organizations having staff complements of 2-5 persons and 6-10 persons were the most 
frequently requesting groups, with 25/34 (76%) organizations in these two categories. 


b) Type of Organization 
Organizations differed in type from incorporated, non-profit, social service agencies, 
to other non-profit human and health service agencies, government, funders, and non- 
incorporated community groups. Table 2.0 provides an overview of this range of 
organization type. 
Table 2.0: Type of Organization 


United Way Funded Agency 


Other Social Service Agency 
Health Services Agency 2 


Other Human Service Agency 3 
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Government or Funder 1 


Non-incorporated Group 


TOTAL 


The greatest number of consulting organizations were non-profit, social service 
providers, with a total of 23/34 (68%) agencies. From this group of participants, 14/34 
(41%) were United Way funded agencies, while 9/34 (26.5%) were funded through some 
level of government or foundations. 


Only 5/34 (15%) agencies were from the non-profit human or health services 
sector. Another 5/34 (15%) non-incorporated community groups made use of the service, 
while only 1/34 (3%) organizations could be described as a government or funder. 


c) Type of Service Delivered by Participating Organizations 


As Table 3.0 indicates, at least half, or 17/34 (50%) of all consulting agencies 
delivered services by means of a case management model. These included 9/34 (27%) 
client counselling agencies, 7/34 (21%) client referral-based agencies, and 1/34 (3%) 
agency delivering long-term care services. 


Table 3.0: Type of Service Delivered by Organizations 


To a lesser extent, 7/34 (21%) agencies delivered services relating to improved 


quality of life and well-being (i.e., preventative or therapeutic programs for clients). A 
further 6/34 (18%) agencies were actively involved in delivering research or planning 
mandates (within their area of service specialization), either as part of their actual service 
delivery, or in the form of specially-funded projects. 


14. 
The remaining 4/34 (11%) agencies and organizations focused service delivery in 
the following area(s): advocacy, funding and allocations, and education or training. 


d) Total Persons per Organization to Consult with ECP 


The actual number of agency personnel, by organization, to consult at some point 
with an ECP consultant, ranged from only 1 person up to 8 persons. Table 4.0 provides 
a description of this breakdown. 


Table 4.0: Total Persons Consulting with ECP (per Organization) 
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TOTAL 


Half of all agencies, or 17/34 (50%) had only one person involved in a direct 
consultation with ECP staff throughout the tenure of the program. Another 7/34 (20%) 
agencies had at least 2 persons consulting with staff, while the remaining 10/34 (30%) 
organizations had between 3-8 different personnel (or volunteers) get involved in some 


sort of direct consultation over the 12-month program. 
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e) Position/Role of First Organizational Contact with ECP 


This variable was considered important, in terms of helping to assess whether or 
not information about the ECP was both relevant to those first contacting the program 
and, to a lesser extent, it might assist program planners in the development of information 
to better meet the needs of these individuals. Based on an analysis, it is evident that the 
information promoting the ECP was primarily received by individuals in administrative 
positions within the organization. This is a reflection of how the ECP mailing list was 
constructed - based on archival data collected through existing SPRC mailing systems 
and on information retrieved from the "Directory of Community Services for Hamilton- 
Wentworth" (2), an annually updated publication of Community Information Services of 
Hamilton-Wentworth (a local non-profit agency). 


Table 5.0 clearly illustrates that the majority of personnel, 28/34 (85%) consulting, 
for the first time with an ECP consultant, were either Executive Directors or Managers. 
Only 6/34 (15%) consulting agencies approached the ECP staff through a different staff 
level or functional area. 

On further investigation, through crosstabulating data pertaining to other agency 
personnel from the same organization consulting with the ECP, it is also evident that a 
majority of secondary program users were also either program managers or co-ordinators. 


Table 5.0: Position/Role of First Contact Person 
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Volunteer (Board Member, etc.) 
TOTAL 
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f) Total Consultation Activities per Organization 


Of 34 agencies consulting with the ECP, almost one-half, or 16/34 (47%), 
consulted only one time with ECP staff. Another 8/34 (24%) had staff who returned to the 
program for a second contact, while the remaining 10/34 (80%) agencies had staff who 
referred to the ECP at least 3 times over the 12-month period. Table 6.0 highlights these 
data results. 


The average number of consultation activities carried out by an ECP consultant, 
in collaboration with another agency, was 2 times during the life of the program. By 
multiplying the number of agencies involved in direct consultation with the number of 
consultation activities undertaken per agency, the total number of ECP direct consultation 
activities was n=76. Thus, on a monthly basis (taken over an 11-month period, allowing 
for holidays) program consultants had a consultation caseload averaging between 7-8 
different direct consultation activities, ranging from one-time information sessions to 
ongoing (longer term) support or technical assistance on some evaluation process. 


Table 6.0: Total Consultation Activities per Organization 


One time consultation 
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g) Nature of Consultation Activities by Number of Organizations Participating 


Based on records data collected by staff to describe the nature of direct 
consultations with community agencies served by the ECP, at least 10 categories of 
involvement can be classified. A description of these categories and the total number of 
agencies (from n=34 total agencies) participating in each of these content/activity areas 
can be found in Table 7.0 below. 


Table 7.0: Nature of Consultations by Number of Participating Organizations 


(n=34 Agencies’) 


epee Art ee 
| Information Systems Management | 24 =] 71.0 
| General Approaches to Evaluation | at | 620 
[Introduction to ECP Serves | = 19 | 880 
ar er Senne oe 
| Research DesignStatises =] 8 
[Implementation of Evaluation Plans] 8 | 
|Special Topics Workshops (in-house) | 8 


* More than one category of activity was possible for each agency 
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The single most frequent request for ECP assistance, with 24/34 (71%) agencies 
in total, was for training and support regarding information or database systems 
management. To almost the same degree, a total of 21/34 (62%) agencies sought advice 
or assistance in understanding general approaches or conceptual designs in evaluation 
planning. A close connection between these two activities, as mutually interdependent 
prerequisites to "comprehensive" planning, is reinforced (at least numerically) by these 
data. Correspondingly, 19/34 (56%) agencies requested more in-depth information and/or 
discussion of the services and program components offered through the ECP, perhaps 
as an opportunity to discuss the complex nature of evaluation and its basic elements of 
information collection and utilization in planning activities. 


More specific and technical "how to" activities, such as measurement techniques, 
instrument design, database management, and research design/analysis ranged in 
participation rate from 15/34 (44%) agencies interested in evaluation measurement 
techniques to 8/34 (23.5%) agencies requesting assistance in research design or 
Statistical applications. 


In terms of providing advice or assistance in the implementation of evaluation plans 
or actual evaluation activities, only 6/34 (17.5%) agencies sought out the services of a 
program consultant. Records indicate that the majority of these agencies were, in fact, 
consulting with the Database Consultant regarding computer-based data management 
systems and, more specifically, these consultations dealt with selecting and utilizating 
software programs to achieve these ends. 


To a lesser degree, ECP facilitated in-house organizational workshops and training 
sessions for requesting agencies. Only 6/34 (17.5%) agencies held topical workshops 
regarding evaluation in their own organizational surroundings, while 3/34 (9%) involved 
the Evaluation Consultant in some sort of board development or training. This outcome 
is possibly a result or reflection of the fact that the agencies involved in direct consultation 
were also, by default, participants in the Evaluation Network (the second component of 
the ECP). A principle function of the Network was, indeed, topical workshops and hands- 
on training, which might explain the fewer number of requests of a direct nature. 
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2. Evaluation Network Outcomes 


Participation in the Evaluation Network was based on voluntary interest in attending 
activities which focused on some topical areas relating to evaluation in human services, 
or in an ongoing interest in collecting resources and information regarding these issues. 
While a great many individuals established themselves as participants as the results of 
general mail-outs promoting the Evaluation Network, a greater number became involved 
by default, as an automatic "benefit" of entering into direct consultation with an ECP 
consultant. In total, n=93 individuals, representing some 76 local agencies or community 
groups were involved as participants at some point during the 12-month development of 
the program. Appendix C provides a list of the organizations with at least one individual 
involved in the Evaluation Network. 


Participation rates for network activities ranged from 5 persons attending a 
specialized “interactive computer workshop" to 145 persons attending a co-sponsored 
workshop on "Evaluating Grant Proposal Development". 


Modes of Evaluation Network Participation 


Based on the key areas of Evaluation Network objectives, identified earlier in the 
Section 4.0: "Structural Overview of ECP Program Development" description, the Network 
was to create and facilitate activities that would enhance the following: information and 
resource-sharing; professional development; community resource development; inter- 
agency collaboration; and community consultation regarding evaluation practices. A 
number of activities were envisioned and carried out, including topical workshops and 
seminars, an educational newsletter, dissemination of network reports and memoranda 
concerning network development, network resource-sharing, and the collection of self- 
reported evaluation data regarding network participation and levels of participant 
satisfaction. 
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a) Workshops and Seminars 


A total of 10 different ECP-sponsored workshops or seminars were held over the 
12-month development of the Evaluation Network. Appendix D provides a list of these 
activities and the total number of participants to attend each event. 


Topics covered a wide range of evaluation issues, including: network development, 
developing management information systems, evaluating database management systems, 
measurement techniques, goal-attainment scaling models, evaluating grant proposal 
development, and a series of (4) interactive computer software workshops. Workshop 
participation ranged in size from 5 individuals to 145 individuals. If these two "outlier" 
figures are dropped from the analysis, an average of 15 individuals attended some sort 
of Evaluation Network workshop or seminar. Also of note, attendance records indicate 
that there was a consistent degree of overlap on the part of some Network participants, 
however, each successive workshop also garnered new attendees. In large part, this may 
be due to the specific and topical nature of these activities. Some participants enjoyed 
an opportunity to meet others and participate in a learning situation on a frequent basis, 
while others may have participated because they had a specific interest or need related 
to that topic. 


b) ECP Newsletter 


A second resource component of the Evaluation Network was a network-wide 
newsletter known as The Evaluator (3). It was intended to be a quarterly report of news 
and activities from around the Evaluation Network and the Hamilton-Wentworth 
community at large. Its focus was intentionally on evaluation issues and contributions 
from Network participants were encouraged by staff. Distribution of the newsletter was 
to all past and current participants listed in the Evaluation Network, however, volunteer 
members of the Social Planning and Research Council also received copies of the 
publication in their monthly mail-outs. 


While the intended circulation was established as four times over the first year of 


ale 
the ECP, actual publication of The Evaluator was limited to a January, 1992 and April, 
1993 mail-out. A number of factors, including an organizational move for the Social 
Planning and Research in the spring of 1993 may have contributed to only two 
publications. This issue will be discussed fully in the Final Evaluation Report. 


c) Network Resource-sharing 


ECP staff attempted to keep Evaluation Network participants abreast of all 
program-related activities whenever possible, through the use of network mail-outs and 
materials to be distributed at workshops and seminars. Notices of meetings and special 
events, follow-up reports and documentation resulting from these activities and general 
information of an evaluative context were sent out to participants whenever budget 
resources and time would allow. 


d) Evaluation of Evaluation Network and ECP in General 


A formal participant survey was administered by mail regarding a description of 
participation in activities, levels of satisfaction with program content and staff service 
delivery, and the identification of strengths and weaknesses deemed inherent in the 
program's development and future directions (refer to Appendix E). This survey was sent 
out post-program to all (n=96) Evaluation Network participants, regardless of whether or 
not they had been involved in any direct consultation with ECP staff. 


Results from this survey were pending at the writing of the Program Summary 


Report, but will become an important consideration in the development of the Final 
Evaluation Report. 


3. Evaluation Consultation Program Resource Development Outcomes 


Resources developed for use by ECP staff and the general public included both 
written materials, such as publications and The Evaluator newsletter, and human 
resources in the form of ECP technical volunteers. 
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Written Resource Materials 


A total of 4 published resource documents were produced as a result of the 
Evaluation Consultation Program (4) (5) (6) (7). These materials were available, for a 
small fee, to participants in the program, its Evaluation Network, and the general public 
as part of the Social Planning and Research Council publications component. 


As noted earlier, an Evaluation Network newsletter, known as The Evaluator, was 
also published at two separate times over the year, with a distribution to over 76 local 
agencies. 


All ECP publications and other staff-collected resources considered of an 
“evaluation and planning" nature have been housed in a separate section of the public- 
accessible SPRC library system. 


Human Resources: Program Volunteers 


Over the course of program development, a number of outside agency personnel 
assisted the ECP staff in developing and facilitating various workshops, seminars, and 
other Network activities. A total of 10 program volunteers were active in this regard at 
some point during program development. In particular, the ECP staff had a long-term 
commitment from 2 highly experienced computer programmers, who were available on 
an as-needed basis, to assist agencies and groups with computer or database requests 
whenever Evaluation Consultant or the Database Consultant had questions from these 
agencies requiring advanced technical input. 
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6.0 Participant Satisfaction with the ECP 


Of n=72 persons who were involved in a consultation with the Evaluation 
Consultation Program over the course of its first year, a total of 30/72 (42.0%) took the 
time to respond to a brief "Participant Follow-up Questionnaire" (refer to Appendix E) after 
program completion. 


1. Participant Response Sample 
The follow-up questionnaire was sent by mail, with an enclosed business reply 

envelope for use in returning the completed survey. Each participant from each agency 
received an individual survey with their own business reply envelope to promote 
confidentiality and ensure that an invidual's responses were mutually exclusive of others 
from the same agency. In addition to receiving the survey, those participants who had 
not returned a survey within a two-week period after mail-out received a reminder 
notification, either by phone or by fax machine. After a three-week period had elapsed, 
it was determined that the receipt of any more surveys was not likely. Thus, staff began 
to prepare and analyze the data for evaluative purposes, with the results noted as follows: 


a) Position of Agency Respondents 

Referring to Figure 2, a total of 6/30 (20.0%) respondents were Executive Directors 
of local agencies, while another 10/30 (83.0%) classified themselves as Agency 
Coordinators or Managers. These two figures combined represented one-half (53.0%) of 
the total respondents to the survey. Since these individuals, by the nature of their 
positions, are the key staff decision-makers in their respective organizations, this balance 
of "decision-makers" to "doers" (the other 47.0% of respondents) suggests that the survey 
sample was fairly reflective of the breakdown of total participants involved with the ECP 
(as previously noted in Section 5.0). 


Of the remaining 14/30 (47.0%) respondents, 4/30 (13.0%) were unit or program 
coordinators, 4/30 (13.0%) were direct service workers, 4/30 (13.0%) were planning or 
research staff, and the remaining 2/30 (7.0%) were administrative support staff. 
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b) How Respondents First Found Out About the ECP 

The single greatest number of respondents, or 11/30 (37.0%), found out about the 
ECP through an initial mail-out which was sent out by program staff in November, 1992. 
Another 7/30 (23.0%) were apprised of the program's offerings by means of direct contact 
with program staff (i.e., the Evaluation Consultant or the Database Consultant). Only 4/30 
(13.0%) found out about the program through another person who had previously been 
involved in one of its activities or in a one-on-one consultation with staff. The remaining 
8/30 (27.0%) learned about the program by means of: an SPRC minor consultation (e.g., 
via contact with another SPRC staff person), reading the general publications of the 
SPRC, through being a volunteer with the SPRC, or as the result of an ECP consultation 
with someone else in their organization. 


c) Most Recent Involvement in the Program 


The distribution of respondents by the most recent contact they had had with 
program staff was fairly even across the 12-month period in which the program was 
offered. Respondents reported their last involvement with the ECP in a range, from 12 
months prior to the program ending (October, 1992), to a status of "current or ongoing" 
(November, 1993). Table 8.0 illustrates this breakdown. 


Table 8.0: Most Recent Involvement with ECP 


| Jan-Mar. 93 | 6 | 
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July-Sept '93 
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2. Participant Satisfaction with Program Components 


Participant satisfaction with the program was measured by means of a 4-point 
rating scale (1=very dissatisfied; 4=very satisfied) as illustrated in the sample survey form 
in Appendix E. Seven program components and related activities were evaluated using 
this scale: 1) telephone consultations, 2) one-on-one consultations with program staff, 3) 
presentations, 4) development of an Evaluation Network, 5) resource development, 6) 
volunteer development, and 7) seminars and workshops. Figure 3.0, "Satisfaction with 
ECP Components" provides an overview of respondent ratings, in chart form, for each 
program component. 


a) Telephone Consultations 

A total of 10/30 (33.0%) survey respondents rated telephone consultations which 
they had undertaken with program staff at some point during the program. The remaining 
20/30 (67.0%) did not apply to this component area. Of the 10 respondents, 9/10 (90.0%) 
Stated that they were mostly to very satisfied with any telephone consultations they had 
experienced with program staff. Only 1/10 (10.0%) respondent was somewhat 
dissatisfied, stating that they had had difficulties contacting staff by telephone (this may 
have been largely due to the inception of a new computerized voicemail system in place 
at the SPRC. 


b) One-on-one Consultations 

Of the 17/30 (57.0%) respondents who rated this component, there were no 
unsatisfied client, and 17/17 (100.0%) were "mostly" to "very satisfied" with the service 
they had received from a program consultant. 


c) Presentations 

This rating area received only 4/30 (13.5%) responses from participants, making 
it difficult to measure satisfaction. However, all 4 respondents rated this aspect of 
program as "mostly" to "very satisfied". 
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d) Evaluation Network 
Of 15/30 (50.0%) participants who had been involved with the Evaluation Network, 
only 1/15 (6.5%) responded that they were "somewhat dissatisfied" with the Evaluation 
Network. The greatest number of participants, or 13/15 (87.0%), were "mostly" to "very 
satisfied" with the development of the Network in its first year. Another 1/15 (6.5%) stated 
that they had been part of the Network, but did not specify a satisfaction rating. 


e) Resources 

Only 10/30 (33.0%) participants rated this item. Of this, 9/10 (90.0%) stated they 
were "mostly" to "very satisfied" with resource development and the resources they had 
received during the first year of the program. The remaining 1/10 (10.0%) person was 
"very dissatisfied" with the range of resources offered by the program. 


f) Volunteer Development 

The total number of program volunteers recruited over the course of the first year 
was 10 persons (refer to Section 5.0: Program Outcome Measures). Of this number, 
6/10 (60.0%) responded to the satisfaction survey. A total of 5/6 (83.0%) volunteers were 
"mostly" to "very satisfied" with their experience or training. Only 1/6 (17.0%) was 
“somewhat dissatisfied" with their involvement. 


g) Seminars and Workshops 

Seventy per cent, or 21/30 respondents had participated in some sort of ECP- 
sponsored workshop or seminar over the course of the year. Of this number, 19/21 
(90.0%) rated their experience as "mostly" to "very satisfied". Only 2/21 (10.0%) were 
“somewhat dissatisfied" with their involvement. 


3. Satisfaction with Program Delivery 


The second area of client satisfaction focused on methods of program delivery 
relative to the program components. Using a similar 4-point rating scale, participants 
were asked to rate their levels of satisfaction with the following program delivery areas: 
ability to educate and train agency personnel, one-on-one consultation skills, knowledge 
of evaluation and planning issues, knowledge of information and database systems, 
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application of computer knowledge, knowledge of survey research techniques, and ability 
to produce resource documents. Figure 4.0 provides a chart describing how participants 
rated the ability of program staff (i.e., Evaluation Consultant, Database Consultant, and 
volunteers) to deliver program services. 


As Figure 4.0 illustrates, with the exception of resource development, most areas 
of service delivery were rated as "mostly" to "very satisfied" for a majority of participants 
responding to scales pertinent to their own experience(s) with the program. Lower levels 
of satisfaction with resource development were expected by program staff, given that 
outcomes for this program area were also lesser than first anticipated at program outset. 


4. Strengths and Weaknesses of the Program in its First Year 


Respondents were asked to complete an open-ended set of questions regarding 
what they considered to be the strengths and/or weaknesses of the program within the 
context of its first year of operation (and given its resource complement). A total of 22/30 
(73.0%) respondents identified strengths of the program, while 16/30 (53.0%) identified 
specific weaknesses as a result of their participation. 


a) Strengths of the Program 


* Knowledge of the program staff in all areas of program content. 

* Availability and accessibility of program staff when requested. 

* Willingness to take time to explain and assist in evaluation development. 

* Provided good information and new ideas about self-evaluation. 

* Staff encouraged others to examine and take part in self-evaluation. 

* One-on-one consultations. 

* Follow-up after initial consultation was supportive and effective. 

* Written materials (from seminars, etc.) were informative. 

* Presentations, seminars, and workshops were well-planned and informative. 
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b) Weaknesses of the Program 


Limited funding did not provide enough oppoytunity for agencies to fully benefit 
from the program beyond Year One (due to time constraints). 

Content was too general for certain agency needs. 

At times, there were not enough staff available to manage all the requests for 
assistance. 

Program content could not possibly cover all the evaluation needs of such a broad 
cross-section of agencies; agencies had too much variance in their levels of 
evaluation experience to contribute meaningfully to any first year planning or 
networking. 

The program was not publicized adequately enough to reach more potential clients. 
Smaller or newly-evolving organizations did not benefit as much from program 
content or format since it did not seem to be sensitive to grass-roots or unique 
approaches to planning. 
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7.0 Summary and Recommendations 


The purpose of this report was to provide a summary evaluation of outcomes 
relating to the development of the Evaluation Consultation Program in its first year of 
operation. The context for this report lies within the goals and objectives outlined as the 
basis for ECP development, both in terms of structural components and program content. 


1. Summary 


In summarizing the findings of Year One, it is important to recognize that the 
original conceptualization of the Evaluation Consultation Program included a Year Two 
and Year Three component. In reducing the three year plan for development and 
implementation of the program to a twelve month period, certain project components were 
limited, in terms of the time and resource development that was considered necessary for 
full and effective implementation. 


An analysis of outcomes relative to the "Consultation Service" component, indicates 
that there has been much interest on the part of agency directors and managers in 
utilizing such a service for specific consultative purposes, the details of which have been 
identified in "Section 5.0: Program Outcome Measures". For a first-time and unique 
program to be developed and implemented in such a fashion as the ECP, and with the 
large number of (n=34) agencies and (n=72) agency personnel requesting assistance, 
there would appear to be an ongoing need for this component of the program to be 
maintained in some constructive form within the community. 


It should be noted that much of the focus for service requested by agency 
managers and other personnel in the first year was regarding "managing information 
systems" and "general approaches to program evaluation". Thus, many of the issues 
which were brought forward to ECP staff were of a conceptual, pre-measurement, and 
planning nature. 


An analysis of the demand for "database" and "computer-based" consultations 
suggests that those organizations seeking assistance and support for these activities were 
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agencies that already had information systems in place, or that were emerging and in 
need of more technical support regarding database management and analysis. It is 
logical to assume that a large percentage of Year One agencies which had sought 
conceptual and pre-measurement input from the Evaluation Consultant would, indeed, 
seek more data collection and management support from the ECP in Years Two and 
Three. 


Participation in the Evaluation Network, as it was conceptualized at an initial 
Network Planning Workshop in December, 1992, was not based on formal membership 
requirements. 


In Keeping with this low-organized approach to development of a diverse network 
of service providers and planners, both participants and ECP staff recognized the 
limitations this would present to the Network's growth in the future, especially after formal 
funding to support such a community network ceased. This factor notwithstanding, it is 
typical (especially in resource-diminished times as these) for new networks of service 
providers to "expand and contract" in size and focus over time. An important consider- 
ation in this regard is the overall nature of the ECP itself, as a developing program of 
consultation in the Hamilton-Wentworth community. 


Resource development remained a component of the ECP which was heavily 
dependent upon input (and output) from Evaluation Network participants and activities. 
The ability of staff to produce topical publications on a regular basis, incorporating the 
time that would be necessary to adequately develop such resources was, to some 
degree,hindered by a shortage of staff time. Future resource production will rely on the 
interest of the Evaluation Network participants and the evaluation research interests of 
agencies like the Social Planning and Research Council to provide more insight. 


Evaluation research has described the first year of any new and innovative program 
as one in which "implementation" measures (like participation rates and levels of program 
enquiry) are considered more valid measures of effectiveness than program outcomes or 
process evaluation. 
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An important consideration in evaluating the Evaluation Consultation Program 
remains some measure of initial participant satisfaction with its components and level of 
delivery. Based on feedback received both formally, through the "Participant Follow-up 
Questionnaire", and informally, in contact between participants and staff, the program was 
rated as "mostly" to "very satisfactory" in terms of content and service delivery. Two key 
areas of weakness identified consistently by participants were: 1) the need for more in- 
depth resources and educational tools for use in agencies requiring evaluative information 
which is immediately accessible and flexible, and, 2) the need for a greater concentration 
of resources (human and material) relating to database development and management. 
The latter need is important to consider in any future development of such a program or 
initiative. It became clear over Year One that the original technical focus of providing 
micro-computer support to agencies as a measured felt need was of secondary concern 
to managers after their requests for support in the development of information manage- 
ment systems that would utilize micro-computers as part of a larger database process. 


2. Recommendations for Future Program Development 


The following recommendations identify key areas of future program development 
within the context of service delivery in Hamilton-Wentworth: 


i There is a continuing need for a program, like the Evaluation Consultation 
Program, to exist in the Hamilton-Wentworth Region. Any future effective 
implementation or continuation of such an initiative should ensure that some 
re-assessment of program goals and objectives is carried out, using 
community participation, and accounting for the feasibility of such a program 
over a longer term than was originally envisioned by program planners. 


Zz Continuation of such a program should take into account the needs of a 
wide variety of agency types and the broad spectrum of experience (of lack 
thereof) as identified through this evaluation. 


3. Critical areas of future program development requiring more resources, 
expertise, or emphasis include: 


a) resources and support in creating and managing adequate information or 
database systems within human service agencies across the Region; 
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b) providing more information and educational opportunities for Agency 
Managers and Executive Directors to learn about emerging practices in the 
program evaluation field (e.g., local training sessions or workshops); 


c) more emphasis on linking these evaluation databases to appropriate tools 
and techniques for analyzing and utilizing them, such as computer software; 


d) more resources directed towards basic research training of agency 
personnel in all areas concerning evaluation and planning techniques, with 
a strong emphasis on data collection, self-directed data management, and 
appropriate analysis procedures, requiring minimal assistance from outside 
consultants; 


e) appropriate funding should be sought to acquire resources and develop 
a unique, user-friendly, and marketable “how-to” software package which 
would function as an evaluation training tool for agency managers and their 
staff. Such a package must be flexible to the needs of a variety of agency 
types and staff resources and function in both a theoretical and interactive 
mode to optimize its utility. 


More emphasis, on the part of the community of providers, for agencies to 
regard evaluation as in important planning prerequisite to effective decision- 
making. A greater degree of commitment must be instilled across all 
agencies. There is a continuing role for the SPRC and other planning 
organizations (including funders) to support agencies in developing 
standardized, valid, and useful approaches to evaluating their services. 


(1) 
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EVALUATION CONSULTATION PROGRAM 
CHRONOLOGY OF KEY PROGRAM ACTIVITIES 
(October, 1992 - October, 1993) 


The following Is an overview of key program activities carried through the Evaluation 
Consultation Program (SPRC) from October, 1992 to October, 1993 Inclusive: 


Funding received from Trillium Foundation. Caroline Ball, an SPRC Core staff person, 
granted a one-year leave of absence to take on the development and coordination of the 
new program until October 3, 1993. 


Initial program development, in the form of mail-outs and faxed announcements to local 
human service agencies, begins to include agency personnel with an interest in 
evaluation issues in the creation of program activities. 


As part of the evolution of carrying out training of staff in evaluation techniques, an 
“Evaluation Network" is proposed to act as a vehicle for group discussion and instruction 
on evaluation practices. The basis for all instructional sessions becomes the four SPRC 
"Evaluation Principles", developed by its Evaluation Task Force earlier in 1991. 


Over 40 different agencies had consulted with the Evaluation Consultant regarding 
specific evaluative concems. Largely as the result of its October publicity campaign, the 
program and its one staff person was beginning to reach capacity in terms of direct 
consultation and agency visitation. 


To help ease the workload, especially with regard to agencies looking for assistance with 
database development, Dale Brown is contracted to act part-time as a Database 
Consultant on an as-needed basis. 


Other community resources who might serve as program volunteers to work, through the 
Evaluation Network, in assisting other agency personnel and in Network activities 
development, are sought by program staff. 


The program acquired the work placement services of a computer programmer/analyst, 
through the Futures Program, to act as a part-time consultant re: computers and 


database development 


Over 65 different community service agencies had been in contact with the Evaluation 
Consultant to enquire about services or to carry out problem-solving activities relating 
to the evaluation of their services. 


January 


February 


The newly-formed Evaluation Network has two separate introductory sessions, hosted 
by the SPRC. A total of 43 agency personnel attended either one or the other session. 
The net result of these meetings was to be a planning meeting in January to short-list 
Network priorities for the first year. 


A Network newsletter, known as The Evaluator, fs published and circulated to all 
persons and/or organizations that had been involved in some way with the ECP to that 
point In time. The Intent of the newsletter was to collect and disseminate Information for 
Network participants interested In evaluation Issues. 


Evaluation Network meets for formal planning session to short-list activities. At least 5 
workshops are identified and 2-3 resource publications. 


Network size Is growing steadily to 50 participants from various Hamilton-Wentworth 
social service agencies. 


The Database Consultants complete 2 separate research initiatives: (1) a Local Area 
Network (LAN) feasibility report, and (2) an in-house assessment of SPRC’s own 


database. 


Planning continues toward a Network workshop on database assessment techniques, 
to be co-facilitated by all three program consultants. 


The active consultation caseload is 20 agencies. 


First ECP newsletter is published, January 28th. 


Two Network workshops are held: 


(1) "Developing Management Information Systems”, facilitated by the Evaluation and 
Database Consultants; 


(2) "Evaluating Database Management Systems”, facilitated by Database Consultant. 


The active consultation caseload remains consistent with 20 agencies requesting 
assistance. 


The number of one-on-one consultations drops by close to 50%, with an active caseload 
of 11 agencies; perhaps due to growth of Network and subsequent workshop activities. 


The program hosts "Measurement Techniques in Evaluation" Workshop; guest facilitators 
include M. Pennock and N. Sidoruk, SPRC. 
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SPRC moves its location to West Avenue School. The net effect on ECP activity is 
minimal re: active consultations. However, due to technical problems on site and some 
space acquisition inadequacy certain Network activities are put temporarily on hold. 


"Goal Attainment Scaling" seminar is hosted by ECP; facilitated by M. Pennock, SPRC. 


v 


Second issue of The Evaluator Is published. 


Over 70 different agency personnel have consulted directly with the Evaluation 
Consultant. 


The ECP begins collaborating with another local network of social/health service 
providers - the Grant Information Group - on a 1/2 day community workshop regarding 
the evaluation of grant proposals. 


A full-day Evaluation Conference is envisioned by staff, to be sponsored by the ECP. 


Ongoing consultations with agencies now total 5-6 agencies; however these 
consultations are time consuming and involve a greater number of ECP resources. 


Preparations are underway with the volunteer computer programmer to facilitate a 4 part 
series on “Interactive Computer Workshops". The focus will be on learning about basic 
softwar applications, including DOS, word processing, spreadsheets, and database 


software. 


The ECP co-sponsors a community-wide “Grant Proposal Development Workshop”. A 
total of 145 participants made this a well-attended 1/2 day workshop. 


As the last series of workshops to be sponsored by the ECP, the Interactive Computer 
Workshops are well underway. At least 20 Network participants had registered for up 
to 4 separate sessions. 
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APPENDIX B - ECP CONSULTATIONS 


Organization/Group 


Beth Tikvah Residence 


Srs. Activ. Main. Pgm. 


United Way Staff 


Volunteer Centre 


ECP Consultations 


Consult Date 
30/12/92 


08/01/93 


04/12/92 


06/01/93 


07/01/93 


/ 193 


I 193 


! 193 


15/09/93 


09/11/92 


-> 


-> 


-> 


-> 


= 


-> 


-> 


Nature 


Discuss Evaluation Planning for 
Year | of Program 


Mtg. with L. Odoardi-Pollard to 
provide technical Input re: 
program measurement 
instrument 


Discuss Principles of 
Evaluation/planning and 
questionnaire development 


Met with Coordinator and 
Volunteer Researcher to 
discuss methodology for 
program eval. of overall agency 


Met with S. Dooley to apprise 
her on ECP and its approaches 


Mtg. with €. Forbes re: 
involvement in ECP Network 
and as a possible “Board 
development and evaluation" 
resource to the ECP 


Mtg. with S. Monte re: role of 
SPRC in training United Way 
agency personnel re: evaluation 
practices 


Mtg. with S. Monte to consult 
re: United Way Allocations 
Principles and Standardized 
methods of accountability from 
member agencies 


Mtg. with S. Monte to provide 
feedback re: “built-in 
evaluation” as part of future 
agency budget submission 
package 


Meeting with Executive 
Director, L. Weaver re: program 
objectives and to set up 
meetings with her project staff 


V.O.N. (Homesharing Program) 


Hamilton Program for Schizophrenia 


St. Martin’s Manor (CFSO) 


G.F.P.0. 


Teen Emotional Health Study 


/ 193 


25/11/92 


29/03/93 


05/04/93 


/ 193 


! 193 


/ 193 


15/04/93 


16/04/93 


20/04/93 


17/09/93 


-> 


-> 


-> 


Mtg. with M. Pennock re: 
translating in-house database 
software to Reflex 
environment; referred to 
Database Consultant 


Initial consultation with J. 
Pawlick, Case Manager re: 
information systems 
management 


Consultation with J. Pawlick, 
Case Manager, re: research 
survey design for client impact 
assessment 


Consultation with M. Oda and 
Paula Forbes (Exec. Dir.) re: 
Goal Attainment Scaling 
techniques 


In-house Consultation re: 
evaluation survey design and 
implementation 


Back and forth facsimiled 
correspondence re: feedback 
on survey development 


Follow-up meeting re: analysis 
and interpretation of survey 
results 


Mtg. with M. Oda and P. 
Forbes re: implementation of 
questionnaire 


R. Jaworsky re: input on 
research methodology 


Mtg. with Researcher re: 
SPSS/PC and database 
management 


Mtg. with Co-ordinator re: 
SPSS/PC export files 


Reg‘! Soc. Services 


North Ham. Comm. Hith Ctr. 


WorkAble Project 


Grant Information Grp. 


Soc. Assist. Reform Workshop 


Adult Basic Educ. Association 


19/07/93 


30/08/93 


07/06/93 


10/06/93 


24/06/93 


14/07/93 


19/07/93 


13/07/93 


05/10/93 


27/09/93 


07/10/93 


“> 


-> 


-> 


-> 


-> 


re: Questionnaire Development 
and computer 
databases 


Mtg. with J. D’Amico/R. 
Mallais re: questionnaire 
development 


Questionnaire development 
meeting with R. Mallais 


Consultation with C. Bian/D. 
Vernon re: Evaluation’ of 
"Access to Opportunity” 
program 


Consultation with M. Keith 
(Health Promotions 
Coordinator) re: ECP and 
general evaluation issues 
related to health promotion 
programming 


Mtg. with Grant Information 
Group re: “Building an 
Evaluation Component into 
Grant Development® 


Consultation with M. Baker re: 
SPSS database development 
and Questionnaire analysis 


Planning meeting re: “Grant 
Proposal Development 
Workshop"; facilitated by 
Evaluation Consultant 


Mtg. to discuss Evaluation 
Form for upcoming workshop 


Half-day workshop, co- 
sponsored by ECP: “Grant 
Proposal Development 
Workshop” 


Facilitating “JobLink” program 
discussion group 


Consultation with L. Babin, L. 
Robinson, and S. Mink (Literacy 
Branch) re: needs assessment 


Second Stage Housing 


Multiple Sclerosis Society 


Scouts Canada 


John Howard Society 


Catholic Youth Organization 


Alternatives for Youth 


YWCA 


06/11/93 


10/11/92 


11/11/92 


13/11/92 


16/11/92 


16/11/92 


24/11/92 


17/11/92 


-> 


-> 


-> 


= 


of H-W Region literacy needs 


Introductory consultation and 
discussion re: computer 
databases with R. Hall 
(Director) 


Meeting with $§. Alyman 
(Director) re: database design 
and software packages; 
referred to Database Consultant 


Introductory meeting with G. 
Guest (Executive Director) to 
discuss their in-house database, 
which was already In place. 


Introductory meeting with B. 
Sibley re: evaluation planning 
practices and basic database 
design Ideas 


Introductory meeting with P. 
Rosser (Program Manager) re: 
in-house database management 
systems; referred to M. Doyle 
(computer programmer on 
placement with ECP) 


Introductory meeting with J. 
Marlin (Exec. Dir.) and S. 
Didtrek re: AY database 
management and 
quuestionnaire design; 
exchange of general research 
materials 


Follow-up meeting to discuss 
program objectives definition 
and development 


Meeting with 3 Branch 
Executive Directors to discuss 


S.1.S.0. 


Labourhood 


SHAIR Resource Centre 


St. Joseph’s Community Hith Ctr. 


Neighbour to Neighbour 


30/04/93 


11/05/93 


02/06/93 


05/07/93 


06/05/93 


14/10/92 


17/10/92 


13/05/93 


21/05/93 


03/06/93 


30/06/93 


-> 


“> 


-> 


-> 


-> 


-> 


Mtg. with D. Brooks (Exec. 
Dir.) and G. DeSantis (SPRC) 
re; building an _ evaluation 
component Into SISO 
activities 


Mtg. with Os. 
Brooks/G.DeSantis, follow-up 
planning meeting for SISO 
Board presentation to be made 
by Eval. Consultant 


Telephone consultation with D. 
Brooks 


Presentation to SISO Board Re: 
"Developing Goals and 
Objectives for Program 
Evaluation” 


Mtg. with P. Randazzo, B. 
House, Andrea Horwath re: 
planning ongoing evaluation 
activities for Labourhood 


Strategic planning preparation 
meeting 


Half-day Strategic Planning 
session; facilitated by 
Evaluation Consultant 


Mtg. with Mary Anne Peters 
(Co-ordinator) re: ECP program 
offerings and SHAIR evaluation 
practices 


Presentation to Research 
Committee re: ECP program 
offerings and SPRC approach to 
evaluation research (Dr. Marilyn 
Craven) 


Mtg. with R. Mallais/J. 
D’Amico re: “Self 
Empowerment Workshops” 
evaluations 


Consultation with N. Williams 


COHR Family Services 


Canadian Hearing Society 


Red Cross 


24/11/92 


25/11/92 


08/12/92 


14/12/92 


Family Violence Prevention Project 17/12/92 


Women’s Centre 


07/03/93 


-> 


-> 


-> 


general focus of ECP and its 
usefulness to YWCA 


Meeting with S. Jewett (Exec. 
Dir) re; database needs and 
general evaluation planning 
concepts for her organization 


Meeting with R. Kendrick 
(Exec. Dir) re: general focus 
and utility of ECP as a program 
vis a vis CHS 


Meeting with S. Mulkewich 
(Manager) re: assisting 
“Community Planning Group" 
with development of 
comprehensive evaluation plan 
and implementation within 
branches 


Formal presentation and 
consultation with Branch 
Managers comprising 
“Community Planning Group" 
re: proposal development for 
evaluation and needs 
assessment of services across 
the region; also provided 
feedback on construction of 
questionnaire aimed at 
distribution in branches across 
Hamilton-Wentworth 


Meeting with D. Mitchell (Co- 
ordinator) to discuss 
questionnaire design 


Consultation with S. Shaw, 
President, re: evaluation 
practices 


APPENDIX C - PARTICIPATING EVALUATION NETWORK AGENCIES 


Lisa Chan 

Settlement and Integration Services Organization 
183 James St. S. 

Hamilton, Ontario 

L8P 3A8 


Ms. Sharon Rupp 

c/o Volunteer Centre of Hamilton-Wentworth 
627 Main St. E., Suite 206 

Hamilton, Ontario 

L8M 1J5 


Ms. Marie Lethbridge 

c/o Volunteer Centre of Hamilton-Wentworth 
627 Main St. E., Suite 206 

Hamilton, Ontario 

L8M 1J5 


Ms. Mary Anne Peters 

Co-ordinator 

SHAIR Intemational Resource Centre 
255 West Ave. N. 

Hamilton, Ontario © 

L8L 5C8 


Ms. Adela Aguilar 

Settlement and Integration Services Organization 
183 James St S. 

Hamilton, Ontario 

L8P 3A8 


Ms. C. Yiu 

Settlement and Integration Services Organiza 
183 James St. S. 

Hamilton, Ontario 

L8P 3A8 


Mr. Joe Rupp 

c/o Volunteer Centre of Hamilton-Wentworth 
627 Main St. E., Suite 206 

Hamilton, Ontario 

L8M 1J5 


Ms. Anneli Tolvanen 

SHAIR International Resource Centre 
255 West Ave. N. 

Hamilton, Ontario 

L8L 5C8 


Ms. Shelley Thompson 

Seniors Activation Maintenance Program 
191 Main St. W., 2nd Floor 

Hamilton, Ontario 

L8P 4S2 


Ms. Janet Millikin 
6-2390 Woodward Ave. 


Burlington, Ontario 
L7R 1T9 


Ms. Susan Chapman 

Executive Director 

Beginnings Counselling and Adoptions 
Services of Ontario Ine. 

1 Young Street, Suite 308 

Hamilton, Ontario 

L8N 1T8 


Mr. Tom Kerr 

Executive Director 

Bold Park Lodge Incorporated 
69 Bold Street 

Hamilton, Ontario 

L8P 1T6 


Ms. Mary Anne Peters 

Coordinator 

SHAIR International Resource Centre 
514-25 Hughson Street South 
Hamilton, Ontario 

L8N 2A5 


Mrs. Martha Pennock 
Coordinator 

VON Homesharing Program 
400 Victoria Ave. N. 
Hamilton, Ontario 

L8L 5G8 


Ms. Peggy Grall 

Canadian Mental Health Association 
Halton Region 

2031 James St. 

Burlington, Ontario 

L7R 1H2 


Ms. Carolyn Heijm 
UNICEF Hamilton 
100 James St. S. 
Hamilton, Ontario 
L8P 2Z3 


Ms. Susan Hall 

Visiting Homemakers Association 
of Hamilton-Wentworth 

c/o St. Elizabeth’s Village 

393 Rymal Rd. W., Suite 302 
Haimtton, Ontario 

L9B 1Y¥2 


Ms. Rosita Hall 

Emergency Shelter Foundation 
Second Stage Housing Program 
135 Rebecca St. 

Hamilton, Ontario 

L8R 1B9 


Ms. Julie Pawlick 

Hamilton Program for Schizophreniz 
102-350 King St. E. 

Hamilton, Ontario 

L8N 3Y3 


Ms. Lynn Edwards R.N. 

Seniors Activation Maintenance 
Program 

191 Main St. W., 2nd Floor 
Hamilton, Ontario 

L8P 4S2 


Mr. Don Mandryk 

Child and Adolescent Services 
P.O. Box 897 

Hamilton, Ontario 

L8N 3P6 


Ms. Sharie Mulkewich 
Canadian Red Cross Society 
Hamilton Branch 

400 King St. E. 

Hamilton, Ontario 

L8N 1C3 


Ms. Nina Truscott 
Manager 

Information Burlington 
2331 New St. 
Burlington, Ontario 
L7R 134 


Mr. Mike Meyer 
Executive Director 
Catholic Family Services 
of Hamilton-Wentworth 
82 Stinson St. 

Hamilton, Ontario 

L8N 182 


Ms. Mae Radford 
Meals-on- Wheels 
414 Victoria Ave. N. 
Hamilton, Ontario 
L8L 5G8 


Ms. Pam Carpenter 
Victorian Order of Nurses 


414 Victoria Ave. N. 
Hamilton, Ontario 
L8L 5G8 


Ms, Paula Forbes 
Catholic Family Services 
of H-W 

82 Stinson St. 

Hamilton, Ontario 

L8N 1S2 


Mr. Ralph DeLuca 
Hamilton Dante Centre 
346 Main St. E. 
Hamilton, Ontario 
L8N 1J1 


Mr. John Dodds 
TranSkills 

201 James St. S. 
Hamilton, Ontario 
L8P 3A8 


Ms. Jan Perry 
Y.W.C.A. 

75 MacNab St. S. 
Hamilton, Ontario 
L8P 3Cl 


Ms. Sandy Shaw 
President 

Women’s Centre of H-W 
80 Augusta St. 
Hamilton, Ontario 

L8N 1R2 


William John 

Volunteer Centre of Hamilton & District 
627 Main St. E., Suite 206 

Hamilton, Ontario 

L8M 1J5 


Elaine McLachlan 

Family Services of Hamilton-Wentworth 
350 King St. E., Suite 201 

Hamilton, Ontario 

L8N 3Y3 


Janet Marlin 
Executive Director 
Alternatives for Youth 
126 James St. S. 
Hamilton, Ontario 
L8P 224 


Dianne Millar 

Hamilton Women’s Detox Centre 
48 Rosslyn Ave. N. 

Hamilton, Ontario 

L8L 7P2 


Ms. Sharon Monte 
Director 
Agency Relations Dept. 


United Way of Burlington, Hamilton-Wentworth 


177 Rebecca St 
Hamilton, Ontario L8R 1B9 


Julie White 

Volunteer Centre of Hamilton & District 
627 Main St. E., Suite 206 

Hamilton, Ontario 

L8M 1J5 


Joanne Tee 

Family Services of Hamilton-Wentworth 
350 King St. E., Suite 201 

Hamilton, Ontario 

L8N 3Y3 


Sue Didtrek 
Alternatives for Youth 
126 James St. S. 
Hamilton, Ontario 
L8P 224 


Ms. Miriam Oda 

St. Martin’s Manor 
Catholic Family Services 
500 Mohawk Rd. W. 
Hamilton, Ontario 

LOC 1X4 


Ms. Lea Odaordi-Pollard 
Beth Tikvah House 

81 Arkell Avenue 
Hamilton, Ontario 

L8S 1N6 


Liz Robinson 

Executive Director 

Adult Basic Education Association 
2 King St. W., Jackson Station 
P.O. Box 57060 

Hamilton, Ontario L8P 4W9 


Ms. Josephine D'Amico 
Neighbour to Neighbour Centre 
28 Athens St 

Hamilton, Ontario 

LOC 3K9 


Neil Williams 

Neighbour to Neighbour Centre 
28 Athens St 

Hamilton, Ontario 

L9C 3K9 


Ms. Carmen Bian 
Social Planner 


Social Planning and Policy Development Division 
Regional Municipality of Hamilton-Wentworth 


P.O. Box 910 
Hamilton, Ontario L8N 3V9 


Leanne Babin 

Adult Basic Education Association 
2 King St. W., Jackson Station 
P.O. Box 57060 

Hamilton, Ontario L8P 4W9 


Ms. Rose Mallais 

Neighbour to Neighbour Centre 
28 Athens St 

Hamilton, Ontario 

LOC 3K9 


Wendy Nelson 
Vice President 


St. Joseph’s Community Health Centre 


2757 King St. E. 
Hamilton, Ontario 
L8G 5E4 


Ms. Heidi Nelson 
Executive Director 
Big Sisters Association 
56 Charlton Ave. W., 
Hamilton, Ontario 
L8P 2Cl 


Ms. Susan Jewett 
Executive Director 
COHR Family Services 
2289 Fairview, 

Unit 208 

Burlington, Ontario 
L7R 2E3 


Ms. Elaine Eastman 

Executive Director 

Halton Social Planning Council 
760 Brant St., Suite 406B, 
Burlington, Ontario 

L7R 4B7 


Mr. Wendell Fields 


Chairperson 

Hamilton Against Poverty 
25 Hughson St. S., 

Suite 506 

Hamilton, Ontario 

L8N 2A5 


Ms. Jill Rumble 

Chief Executive Director 

Young Women’s Christian Association 
75 MacNab St. S., 

Hamilton, Ontario 

L8P 3Cl 


Ms. Janet Marlin 

Ms. Sue Didtrek 
AY/Alternatives for Youth 
126 James St. S., 
Hamilton, Ontario 

L8P 2ZA 


Mr. Andrew Murie 
Executive Director 

Big Brother Association 
639 Main St. E, 
Hamilton, Ontario 
L8M 134 


Mr. G. A. Guest 
Executive Director 
Boy Scouts of Canada 
375 James St. S., 
Hamilton, Ontario 
L8P 3B9 


Mr. Robert Kendrick 
Executive Director 
Canadian Hearing Society 
2301 King St. E, 
Hamilton, Ontario 

L8K 1X6 


Mr. Peter Rosser 

Catholic Youth Organization 
90 Stinson St, 

Hamilton, Ontario 

L8N 1S2 


Mr. Rob Hart 

Community Information Services 
P.O. Box 2700, Station "A", 
Hamilton, Ontario 

L8N 4E4 


Ms. Judy Mintz 

Executive Director 

Dundas Community Services 
10 Market St. S., 

Dundas, Ontario 

L9H 5G4 


Ms. Elaine McLachlan 

Ms. Joanne Tee 

Family Services of 
Hamilton-Wentworth 

350 King St. E., Suite 201, 
Hamilton, Ontario 

L8N 3Y3 


Mr. Brian Sibley 
Manager, Direct Services 
John Howard Society 

of Hamilton-Wentworth 
128 Hughson St. N. 
Hamilton, Ontario 

L8R 1G6 


Ms. Sandy Alyman 

Executive director 

Multiple Sclerosis Society, 
Hamilton & District 

107 George St., 

Hamilton, Ontario 

L8P 1E3 


Ms. Jill Elliott 

Branch Director 

Ottawa Street Community 
Y.W.C.A. 

52 Ottawa St. N., 

Hamilton, Ontario 

L8P 3Y7 


Ms. Ines Rios 
Immigrant & Settlement 
Services of H-W 

332 James St. N. 
Hamilton, Ontario 

L8L 1H2 


Ms. Liz Weaver 

Executive Director 

The Volunteer Centre of 
Hamilton & District 

627 Main St. E., Suite 206, 
Hamilton, Ontario 

L8M 1J5 


Ms. Birgitt Bolton 
Executive Director 
Wesley Urban Ministries 
75 Mary St. N., 
Hamilton, Ontario 

L8R 1K2 


Ms. Kathleen Outwater 
President 

H-W Stroke Recovery 
Association 

3-500 Upper Wellington Street 
Hamilton, Ontario 

L9A 3P5 


Ms. Jean Potter 

Executive Director 

Canadian Diabetes Association 
501 Concession Street 
Hamilton, Ontario 

L9A ICl 


Ms. Sheila Hagens 
Volunteer Centre 

627 Main St. E., 2nd Floor 
Hamilton, Ontario 

L8M 1JS 


Ms. Donna Imeson 
Executive Director 
Participation House 


2080 Trinity Church Rd., R.R.#1 


Binbrook, Ontario 
LOR 1C0 


Ms. Joan Nelson 
Participation House 


2080 Trinity Church Rd., R-R.#1 


Binbrook, Ontario 
LOR 1C0 


Ms. Karen Craig 
Living Rock Ministries 
P.O. Box 83015 
Hamilton, Ontario 
L8L 8E8 


Ms. Dianne Millar 
Women’s Detox Centre 
48 Rosslyn Ave. N. 
Hamilton, Ontario 
L8L 7P2 


Ms. Shelley McCarthy 
Catholic Family Services 
of Hamilton-Wentworth 
82 Stinson St. 
Hamilton, Ontario 

L8N 1S2 


Ms. Marg Kowalski 
Wesley Urban Ministries 
75 Mary St. 

Hamilton, Ontario 

L8R 1K2 


Mr. Bill MacKinnon 
Wesley Urban Ministries 
75 Mary St. 

Hamilton, Ontario 

L8R 1K2 


Ms. Myrtle Greve 

VON 

Community Development 
414 Victoria Ave. N. 
Hamilton, Ontario 

L8L 5G8 


Ms. Janis North 

VON 

Visitng Nursing Program 
414 Victoria Ave. N. 
Hamilton, Ontario 

L8L 5G8 


Ms. Paula Forbes 

St. Martin’s Manor 
Catholic Family Services 
500 Mohawk Rd. W. 
Hamilton, Ontario 

LOC 1X4 


Mr. Tony Quick 
Catholic Family Services 
82 Stinson St. 

Hamilton, Ontario 

L8N 1S2 


Ms. Lisa Paladino 

153 National Dr. 
Stoney Creek, Ontario 
L8G 5B3 


Mr. Dan Gerrard 
Hamilton-Wentworth Star 
2 Oriole Crescent, Unit 41 
Hamilton, Ontario 

L8H 6G8 


Ms. Sarah Keates 
Hamilton-Wentworth Star 
2 Oriole Crescent, Unit 41 
Hamilton, Ontario 

L8H 5G8 


Ms. Lisa Naunder 
Canadian Hearing Society 
2301 King St. B, 
Hamilton, Ontario 

L8K 1X6 


Ms. Elaine Forbes 
Coordinator 

Leadership Development 
Services, The United Way 
177 Rebecca St. 

Hamilton, Ontario 

L8R 1B9 


Ms. Marg Kowalski 
Multiculturalism and 
Citizenship Canada 

150 Main St. W., Suite 425 
Hamilton, Ontario 

L8P 1H8 


Ms. Lorraine Chapman 
Director 

Hamilton Women’s Detox 
Centre 

48 Rosslyn Ave. N. 
Hamilton, Ontario 

L8L 7P2 


Volunteers: 


Mr. Dan Abbott 
46 Bemard St. 
Hamilton, Ontario 
L8H 6C7 


Dr. Richard Marlin 


Medical Rehabilitation Service 


25 Charlton Ave. E. 
Hamilton, Ontario 
L8N 1Y2 


Ms. Rosemary Foulds 
Social Planning Division 
Social Services Dept. 
P.O. Box 910 
Hamilton, Ontario 

L8N 3V9 


Ms. Wendy Nelson 
Vice President 

St. Joseph’s Community 
Health Centre 

2757 King St. E. 
Hamilton, Ontario 

L8G SEA 


Teen Emotional Health Study 


c/o Ms. Debbie Jones 
CH-EVEL-2 


Chedoke McMaster Hospital 


Box 2000, Station A 


Ms. Denise Brooks 
Executive Director 
Settlement & Integration 
Services Organization 
183 James St. S. 
Hamilton, Ontario 

L8P 3A8 


Ms. Mary Keith 


North Hamilton Community 


Health Centre 
$54 John St. N. 
Hamilton, Ontario 
L8L 4S1 


Ms. Paula Randazzo 


Labourhood Homes Resource 


Centre Inc. 

350 Kenilworth Ave. N. 
Hamilton, Ontario 

L8H 4T3 


Ms. Betty Ann Hill 
Nelson Youth Centre 
4225 New St. 
Burlington, Ontario 
L7L 1T3 


Ms. Dorothy Bartolos 
The Hamilton Foundation 
Standard Life Centre 
205-120 King Street West 
Hamilton, Ontario 

L8P 4V2 


APPENDIX D - EVALUATION NETWORK: WORKSHOPS/SEMINARS 


Evaluation Network: List of Workshops and Seminars 


(1992 - 1993) 
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APPENDIX E - ECP PARTICIPANT FOLLOW-UP QUESTIONNAIRE 


EVALUATION CONSULTATION PROGRAM 


Participant Follow-up Questlonnalre 


A._ Participant Information 


4. 


a) 


Rating Scale: 


Agency/Organization: 


Please describe the type of job or position you hold within your agency/organization by placing 
a check () in the appropriate box 


Name: 


Executive Directer () Agency Coordinator! () Unit/Program () 
Manager Coordinator 

Administrative () Direct Service () Planning/ () 

Support Staff Worker, Counsellor, Research/ 
Advocate, etc. Projects 

Other () [Please Describe]: 


If possible, please identify the most recent month (effective October, 1992 - October, 1993) that 
you were involved in an Evaluation Consultation Program activity or personal consultation with 


a staff person: 
Month: Year: 19_ 


Please describe how you first found out about the Evaluation Consultation Program: 


In which of the following ways did you participate or get involved with the Evaluation Consultation 
Program (Please check ( ) as many categories as appropriate); 


b) For each of the activities you have identified, please rate your general level of satisfaction with that 


specific activity by circling the appropriate response category: 


1 = Very Dissatisfied; 2 = Somewhat Dissatisfied; 3 = Somewhat Satisfied; 
4 = Very Satisfied 


() Telephone consultation (with Evaluation/Database Consultant) 
1 2 3 4 
() One-on-one meeting (with Evaluation/Database Consultant) 


1 2 3 4 


ey 


() 


() 


() 


() 


() 


2. 


Meeting or special event in your own organization (e.g., an in-house evaluation 
activity or workshop) where Consultant assisted/facilitated 


1 2 3 4 


Participation in Evaluation Network (sponsored by Evaluation Consuttation 
Program) 


1 2 3 4 

Resource development (publications, ECP newsletter, etc.) 

1 2 3 4 

As a volunteer resource to the program (e.g., facilitator, instructor...) 
1 2 3 4 


Attendance/participation in an Evaluation Consultation Program-sponsored 
seminar, workshop, special presentation, etc. 


1 2 3 4 


Other, please describe: 


1 2 3 4 


Based on your involvement with the Evaluation Consultation Program, please rate your level of 
Satisfaction with program staff (i.e., Evaluation Consultant/Database Consultant) in the following 


areas by circling the appropriate response: 


Rating Scale: 


1 = Very Dissatisfied; 2 = Somewhat Dissatisfied; 3 = Mostly 
Satisfied; 4 = Very Satisfied; n/a = not applicable 


a) Assisting and educating groups, organizations, and agencies in the identification, 
evaluation and selection of appropriate methods of information management, 
client/program assessment, and evaluation. 


1 2 3 4 Wa 


b) Providing individualized consultation to groups, organizations, and agencies as requested 
regarding all aspects of evaluation planning and measurement. 


1 2 3 4 Wa 


6 a) 


b) 


c) 


Knowledge in the following area(s): 


Evaluation Planning (e.g. Identifying outcomes/process-based objectives, criteria 
for measurement of organizational/program objectives, etc.) 


1 2 3 4 n/a 
Information/database management systems development 

1 2 3 4 nla 
Micro-computer applications and/or technical support 

1 2 3 4 nla 
General survey and participatory research approaches, Including questionnalre 
construction, administration, analysis, and Interpretation of results 

1 2 3 4 n/a 
Identification and/or development of resources for use by groups, organizations, 
or agencies In planning for evaluation activities (e.g., “A Primer for Managing 
Information Systems In Non-Profit Organizations", Implementation of a 4-week 


“Interactive Computer Sessions” series, etc.) 


1 2 3 a Wa 


Briefly describe what you consider to be strengths of the Evaluation Consultation Program: 


Briefly describe what you consider to be weaknesses of the Evaluation Consultation Program: 


Thank you for your co-operation In the completion of this follow-up survey. Your Input 
Is essential In the evaluation of this program. Please ensure that this form 's returned In 


the enclosed envelope NO LATER THAN FRIDAY, DECEMBER 10TH, 199 
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